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Description 

METHODS AND SYSTEMS FOR TESTING COMMUNICATIONS 
NETWORK COMPONENTS 

5 Related Applications 

This application claims the benefit of United States Provisional Patent 
Application Number 60/243,691 filed October 27, 2000, the disclosure of 
which is incorporated herein by reference in its entirety. 

10 Technical Field 

The present invention relates to the testing of a communications 
network. More particularly, the present invention relates to methods and 
systems for automated, integrated testing of multiple devices in a 
communications network. 

15 

Background Art 

Wireless and wireline communications networks have evolved to 
encompass a variety of services, switching platforms, applications, data 
processing system hardware and equipment that are collectively referred to 

20 as network products. Before deploying such network products within a live 
network environment, it is important to first test these network products to 
ensure that their operation is as expected. Typical telecommunications 
environments rely upon network elements (e.g., packet switches, database 
nodes, routers, etc.) produced by a variety of manufacturers, and such 

25 network elements may each further include a number of operating system or 
application software revisions. Consequently, one of the primary objectives 
or goals of the testing process prior to deployment is to identify and resolve 
any potential problems in the provisioning of network services that may 
result from the diverse nature of the collection of network elements. Such 

30 problems may include, for example, compatibility problems between newly 
developed network elements and existing or legacy equipment/software. 
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A complete test regimen may involve multiple test platforms that are 
required to perform several tests in parallel on several switches, with a 
variety of resources being required to perform each test. For example, test 
platforms may require connections to several switches in order to send 
5 commands corresponding to a test, and to subsequently collect results of the 
test. From an operational standpoint, providing a dedicated connection from 
each test platform to each switch is typically not practical due to the large 
number of dedicated connections that would be involved. Furthermore, a 
particular test platform may not be capable of being physically connected to 

10 multiple switches at the same time. 

As an alternative to simultaneous permanent connections, a 
temporary connection may be manually established from a test platform to a 
device under test (DUT) when the test platform needs to communicate with a 
test program executing on the DUT. However, such manual connection 

15 techniques are time consuming and inefficient, and consequently are often 
deemed unacceptable by network operators. 

With particular regard to test platforms, it will be appreciated that such 
equipment can be driven by a data processing system software application 
that communicates with the DUT via commands over a serial connection. A 

20 problem with such test devices is that they typically require a command line 
interface dedicated to accomplishing a narrowly defined test objective and is 
therefore both cumbersome and time consuming to use. Scripted batch files 
that automate a series of commands from a test platform to the DUT can be 
used to alleviate this problem, as the use of batch files does not require 

25 manual entry of commands to the connected test device. Consequently, a 
single batch file, or small set of batch files, which contains all the commands 
necessary to accomplish a particular test objective may be employed to 
execute a complicated sequence of test instructions, thereby freeing the test 
operator to perform other tasks for duration of the test sequence. However, 

30 batch files must still be written using device-specific instructions and require 
that the operator have detailed knowledge of the command line interface 
commands of each device being tested. 
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Figure 1 is a schematic diagram illustrating a conventional 
architecture associated with the testing of a telecommunications packet 
switch. Figure 1 includes a test architecture, generally indicated by 
reference numeral 100, that is comprised of a test management platform 
5 (TMS) 110, a signal transfer point (STP) 112, a manually-operated STP 
administrative interface 114, a network test message generator/monitoring 
node 116 (e.g., the Tekelec MGTS™ testing and monitoring product), and a 
manually-operated administrative interface 118. It will be appreciated that 
STP 112 and MGTS™ 116 are directly connected via dedicated signaling 

10 system 7 (SS7) signaling links 120. MGTS™ 116 generates SS7 signaling 
message traffic and simulates SS7 signaling nodes that could be connected 
to STP 112 in a real network. TMS 110 and the two administrative interface 
terminals 114 and 118 are connected to STP 112 and MGTS 116 nodes via 
RS-232 links. More particularly, STP administrative terminal 114 and 

15 MGTS™ administrative terminal 118 are connected to the command line 
interfaces of STP 112 and MGTS 116, respectively. TMS 110 is also 
connected to the command line interfaces of both STP 112 and MGTS 116. 

As such, it will be appreciated that a complex test scenario could 
involve a large degree of operator intervention and subsequent manual 

20 configuration of the two devices under test. For instance, a particular test of 
STP 112 may require that MGTS™ 116 attempt to bring one of the plurality 
of SS7 communication links 120 into service. Execution of this portion of the 
test can be initiated by TMS 110 via one or more commands that are sent to 
the MGTS node via the TMS-to- MGTS™ command line interface 

25 connection. A particular test scenario being executed by the TMS 110 could 
require a dynamic or decision-tree-type response on the part of the TMS 
(i.e., selection of the next test is dependent on the result of the preceding 
test). For instance, TMS 110 may be required to perform a second test, 
TestCase_AlignedTrue if the particular SS7 signaling link in question was 

30 successfully aligned. If the signaling link in question was not successfully 
aligned, TMS 110 may be required to perform a third test, 
TestCase_AlignedFalse. As such, a human operator is required to observe 
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the results of the first test via the MGTS™ administrative terminal 118, and 
provide feedback to TMS 110. Such manual feedback is human-resource- 
intensive, inefficient and prone to operator error. 

With regard to overall test tool evolution, the incorporation of graphical 
5 user interface (GUI) based enhancements to testing systems have been 
instrumental in making the testing process more intuitive and user-friendly. 
However, such enhancements have not focused on the human operator- 
intensive elements of the testing process, such as reducing the need for 
device-specific commands and different terminals connected to each device 

10 being tested. 

Local area network (LAN) technology and communication 
architectures have also enhanced the testing of network products. GUIs are 
conveniently located, either locally or remotely, to a particular data 
processing system that drives the actual testing of network products. 

1 5 Client/server technology is utilized to provide remote access to test systems. 
A client application includes a GUI that allows a user to access a remote test 
system. A server application receives and processes requests from the 
client. The server also executes on a system in the network. The 
client/server framework allows a client to be located on any system in the 

20 network, even on the same system on which the server resides. 

With the many varieties of network products and methodologies for 
testing of network products currently available, test devices must be 
proficient in testing a large number of applications and executing the many 
varieties of test cases associated with those applications. Thus, a test 

25 device must be capable of learning or adapting to new test case formats and 
new test interfaces as required. 

A test case defines how a particular test will be performed. For 
example, a test case may simply be a batch file containing instructions for 
performing a test, or a test case may be a file of commands or directives 

30 administered through a GUI. Additionally, a test case may include a data 
structure stored in memory that is built by a data processing system upon 
receipt of instructions from a client or other application. Regardless of how a 
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test case is implemented the, a test case embodies the action which will take 
place in order to perform a test. 

From an operational perspective, there exists a need for the capability 
for users in a network environment to share test cases and test case results 
5 for multiple devices under test in a test network environment. There also 
exists the need for a testing system that performs any type of test case 
without knowing the specific test case formats and methodologies of each 
and every test case available for execution on a test network. 

Additionally, there exists a need for the capability of designating the 

10 test functions to be tested as a sequence of interrelated, cross- 
communicated test functions. As discussed briefly above with regard to the 
example shown in Figure 1, values produced as output by a first DUT may 
be used as input by a second DUT. However, as indicated above, using the 
results from one test case as input to another test case requires manual 

15 intervention by an operator with detailed knowledge of device-specific test 
case commands. Accordingly, there exists a long-felt need for methods and 
systems for testing communications network components that reduce the 
need for operator intervention and knowledge of device-specific test 
commands. 

20 

Disclosure of the Invention 
The present invention includes methods and systems for creating and 
executing sequences of interrelated test cases and providing a generalized 
test environment that allows complete automation of test cases. The test 

25 cases are independent of the number or types of devices under test. As 
such, a test operator responsible for writing a test script need not know the 
device-specific commands because test environment device packages map 
the device-specific commands to a common script language. Therefore, the 
operator need only be familiar with a common script language rather than 

30 device-specific test commands for multiple devices under test. Furthermore, 
the present invention provides a common interface that allows interaction 
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between multiple devices under test, thereby achieving the desired system 
performance described above. 

According to one aspect, the present invention includes a 
communications network testing system that facilitates autonomous or 
5 attendant-free interaction between the administrative interfaces of multiple 
network devices under test. The present invention includes an abstract 
command language (ACL) interface that enables cross-communication 
between dissimilar devices (i.e., devices that employ different administrative 
interface protocols). As such, the results of a first test on a first device under 

10 test may be used to automatically trigger the reconfiguration of a second 
DUT and subsequent execution of a second test. Furthermore, the second 
test may utilize information returned from the first test on the first DUT. 
Again, it is particularly significant that the first DUT triggers a reconfiguration 
of the second DUT, even though the first DUT and second DUT may employ 

15 different administrative interface protocols. Such functionality allows 
complex test sequences that involve interaction among multiple DUTs to be 
easily automated and remotely executed without requiring a significant 
degree of human test operator intervention. The abstract command 
language interface has a layered architecture that includes a tool command 

20 language (TCL) object-based interface layer. This TCL object-based layer 
utilizes the abstract command language to facilitate communication with 
various independent testing and switching equipment via access to their 
respective administrative command line interfaces. A unique, device-specific 
communication interface package is included for each type of device under 

25 test. 

According to another aspect, the present invention includes a fully 
integrated test case and test plan editor, where test plans and their 
associated test cases are maintained in a version-controlled environment. 

According to yet another aspect, the present invention includes an 
30 arbitration engine that dynamically allocates resources when users attempt 
to simultaneously schedule test cases that require the same resources. 
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The version-controlled environment may be extended to include 
operating software associated with each DUT. As such, a test plan may 
specify a particular version of a test case, which may in turn specify a 
particular program or software version that is to be installed on a given DUT 
5 for the duration of the test. Corresponding test results may also be stored 
and maintained in the-version-controiled environment. 

The present invention further includes a graphical user interface that 
may be accessed and operated via a wide area network or local area 
- network connection. This GUI simultaneously displays test plan, test case, 
10 and corresponding test results. The GUI is further adapted to provide 
access to the test plan/test case editor, as described above. 

Accordingly, it is an object of the present invention to provide a test 
system that facilitates real-time, inter-DUT communication during execution 
of a test without requiring human test operator intervention. 
15 It is another object of the present invention to provide communication 

interface packages for mapping abstract command language commands to 
device-specific commands so that a test case designer is not required to 
have specific knowledge of device-specific commands. 

Some of the objects of the invention having been stated hereinabove, 
20 other objects will become evident as the description proceeds, when taken in 
connection with the accompanying drawings as best described hereinbelow. 

Brief Description of the Drawings 
Figure 1 is a schematic diagram illustrating a conventional test system 
25 architecture. 

Figure 2 is a schematic diagram illustrating a test system according to 
an embodiment of the present invention. 

Figure 3 is a schematic diagram illustrating a test environment that 
utilizes a test system according to an embodiment of the present invention. 
30 Figure 4 is a block diagram illustrating a test management system 

client and version control system according to an embodiment of the present 
invention. 
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Figure 5 is a block diagram illustrating exemplary contents of a 
communication interface package according to an embodiment of the 
present invention. 

Figure 6 is a block diagram illustrating an execution manager - 
5 according to an embodiment of the present invention. 

Figure 7 is a flow chart illustrating exemplary functional processes or 
steps associated with test system operation according to an embodiment of 
the present invention. 

Figure 8 is a block diagram illustrating exemplary test management 
10 system input and output messages according to an embodiment of the 
present invention. 



Detailed Description of the Invention 
Figure 2 is a simplified schematic diagram of a test system according 

15 to an embodiment of the present invention. In Figure 2, test system 200 
includes a test tools server 210, a browser 212, a test management system 
client 214, and a test controller 215, all of which are connected via network 
216. Server 210, browser 212, client 214 and controller 215 may each 
execute on general purpose computing platforms. Network 216 may be a 

20 private wide or local area network (i.e., corporate intranet) or a public WAN, 
such as the Internet. 

Test tools server 210 maintains a library of device-specific 
communication interface packages, where each device-specific 
communication interface package contains information for mapping 

25 executable instructions between a common or abstract command language 
and the command line interface language of a specific device. Test tools 
server 210 may also maintain one or more generic communication interface 
packages that provide certain basic, core communication interface 
functionality that is device independent. Exemplary communication interface 

30 packages will be discussed in detail below. 

In one embodiment, test tools server 210 maintains the 
communication interface package library in a version-controlled environment. 



WO 02/056541 



PCT/US01/48677 



-9- 

such as that provided by the CLEARCASE™ available from Rational 
Software Corporation or similar version control system. As such, multiple 
versions of a single device-specific communication interface may be 
maintained in a secure and version-controlled manner. 
5 In addition to maintaining a communication interface package library, 

test tools server 210 may also maintain a library of test plans and the 
associated individual test cases used in formulating test plans. In one 
embodiment, such test cases may include data files containing tool control 
language and abstract command language instructions. In any event, such 

10 test plan and test case information is maintained in a version-controlled 
environment, in much the same manner as the communication interface 
package library described above. Following the execution of a test plan or 
individual test case, test tools server 210 maintains the associated test 
results in a test results library that preferably also resides in a version- 

15 controlled environment. As such, all aspects of a particular test scenario 
(e.g., communication interface packages, test plans, test cases, and test 
results) are maintained in a version-controlled environment, which 
significantly enhances the quality control and auditing process associated 
with communication component and system testing. 

20 Browser 212 allows a user to simultaneously view test plans, test 

cases and corresponding test results associated with various systems of 
devices or individual devices under test. As described above, such 
information is maintained in a version-controlled environment on test tools 
server 210 and is accessible via communications network 216. Browser 212 

25 may execute on a disk operating system (DOS), WINDOWS®, or Unix-based 
computing platform Examples of browsers suitable for use with embodiments 
of the present invention include Netscape NAVIGATOR® or Microsoft 
INTERNET EXPLORER®. A plurality of browsers can be simultaneously 
connected to test tools server 210 via communications network 216 for 

30 simultaneous viewing of test cases and test case results. In the case where 
communications network 216 corresponds to a public WAN, such as the 
Internet, browsers 212 may execute on computers geographically located at 
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great distances from the actual test laboratory where the physical test 
equipment is located. A test system of the present invention facilitates the 
testing of a communications system that includes multiple DUTs where the 
DUTs are not physically co-located. 
5 Test management system client 214 communicates with test tools 

server 210 via network 216 and allows a user to perform a number of 
activities associated with test system operation Including: scheduling test 
plan execution, constructing and editing of test plans, and constructing and 
editing test cases. Because the test plan, test case, and communication 

10 interface package information is maintained in a version-controlled 
environment on test tools server 210, multiple test management system 
clients or client sessions may be simultaneously active and in 
communication with test tools server 210 without the risk of operator 
confusion due to version problems. 

15 Figure 3 illustrates a test tools server 210, a test management system 

client 214, and a test controller 215 connected to multiple devices under test. 
In one embodiment, such devices under test may include, but are not limited 
to: a service control point (SCP) 218, a home location register (HLR) 219, a 
visitor location register (VLR) 220, a short message service (SMSC) 221, a 

20 domain name system (DNS) server 222, a data packet router 224, a 
signaling gateway (SG) 226, a network simulator and monitoring device 228, 
a signal transfer point 229, and a Softswitch (SS) node 230. The 
communication nodes listed above are well known to those skilled in the art 
of telecommunication and data networks, and as such will not be described 

25 in detail herein. A detailed description of the illustrated signaling system 7 
(SS7) and IP communications devices (e.g. SCP 218, SG 226, STP 229, 
and Softswitch 230) can be found in Russel, Travis, Signaling System #7, 
McGraw-Hill, Third Edition, 2000. A detailed description of the illustrated 
wireless communication network elements (e.g. HLR 219, VLR 220, and 

30 SMSC 221) can be found in Mouly, M. and Pautet, M. B., The GSM System 
for Mobile Communications, Cell & Sys, First Edition, 1992. Data 
communications network elements 222 and 224 are described in Stevens, 
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Richard, TCP/IP Illustrated, Volume I: The Protocols, Addison-Wesley, 
1994. 

In order to test one or more devices, test management system client 
214 accesses a test plan and associated test case files from test tools server 
5 210 and requests that the test cases be executed by test controller 215. 
Test controller 215 establishes connections with the devices under test and 
executes the test cases received from test tools server 210. Results 
associated with a particular test plan or test case are then provided to test 
tools server 210 for storage in the version-controlled environment. 

10 According to an important aspect of the invention, test tools server 210 
includes an arbitration engine 231 that dynamically allocates resources when 
users attempt to simultaneously attempt to use the same resources. The 
automated test system architecture according to the present invention allows 
multiple users in multiple locations to utilize the same test system resources. 

15 For example, a test system user in Richardson, Texas and a test system 
user in Morrisville, North Carolina may each desire to test software on STP 
229, which is located in a lab in Morrisville, North Carolina. Both users may 
access test tools server 210 using TMS clients 214 and schedule tests to be 
executed. Test tools server 210 may include a static scheduler that allows 

20 the users to select time slots for test execution in an attempt to avoid 
scheduling conflicts. However, because test cases can run longer than 
expected, one user's test may conflict with the time slot scheduled for 
another user's test. Arbitration engine 231 dynamically allocates resources, 
in this case, STP 229, so that the test cases do not conflict with each other. 

25 In this example, if user A's test case is nearly complete, arbitration engine 
231 may allow the test to complete and dynamically re-schedule the test 
requested by user B. Exemplary dynamic resource allocation algorithms that 
may be used by arbitration engine 231 include random allocation, first in, first 
out, priority scheduling, etc. Using one or more of these resource allocation 

30 algorithms, arbitration engine 231 dynamically resolves resource conflicts 
before they occur. 
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As illustrated in Figure 3, test controller 215 is capable of 
simultaneously communicating with multiple DUTs. Test contro!ler215 also 
facilitates autonomous or attendant-free interaction between the 
administrative interfaces of multiple network devices under test. As such, 
5 the results of a first test on a first device under test may be used to 
automatically trigger the reconfiguration of a second device under test and 
the subsequent execution of a second test, where second test may utilize 
information returned from the first test on the first device under test. 

It is an important aspect of the invention that test controller 215 allows 

10 a first DUT to trigger a reconfiguration of a second DUT, even though the 
first DUT and the second DUT may be completely different types of 
communication devices with different administrative or command line 
interfaces. Such functionality allows complex test sequences that involve 
interaction among multiple DUTs to be easily automated and remotely 

15 executed without requiring a significant degree of human test operator 
intervention. For instance, in the case of SS7 telecommunications related 
devices that utilize the message transfer part (MTP) protocol, complete MTP 
level two or level three conformance testing can be accomplished in a matter 
of hours with little or no human-operator intervention. Previously, such 

20 comprehensive testing was human-operator-intensive and could take one or 
more weeks to complete. 

Figure 4 is block diagram of one embodiment of a test system of the 
present invention including a TMS client 214, a test controller 215, a system 
of devices under test 302, and a test tools server 210. In this embodiment, 

25 TMS client 214 includes a graphical user interface 310, a test plan/test case 
editor 314, , and a reporting manager 318. Test controller 215 includes a 
test plan and test case execution manager for executing test cases received 
from test tools server 210. GUI 310 provides a human operator with an 
easy-to-use, icon-driven interface to the overall test system of the present 

30 invention. GUI 310 enables an operator with minimal knowledge of the 
devices under test and their respective administrative interface protocols to 
perform a complex battery of integration and system performance tests. 
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Test tools server 210 illustrated in Figure 4 includes a version control 
environment 350, such as CLEARCASE™. A number of database 
elements, including a test plan/test case library 352, a test results library 
354, a communication interface package library 356, and a device under test 
5 operating software library 358, reside within version-controlled environment 
350. Individual software files (e.g., source code, compiled code, scripts, 
data, etc.) are stored in such an environment so that user access to and 
manipulation of the files may be controlled and monitored. 

According to an important aspect, the present invention includes a 

10 plurality of communication interface packages. A communication interface 
package is a software entity that defines the mapping between device- 
specific command line interface commands and common or device- 
independent abstract command language commands. In a preferred 
embodiment, communication interface packages include classes having 

15 functions and data structures used to test specific devices. Examples of 
mapping functions that may be included in a communication interface 
package are shown in Figure 5. In Figure 5, communication interface 
package 370 includes mapping information associated with a signaling 
gateway (SG). The term "signaling gateway," as used herein, refers to a 

20 network element capable of routing both SS7 and IP messages. In this 
particular example, an abstract command language command 
"lsLinkAligned(L1,1) M that is related to the alignment status of a 
communication link is mapped to the corresponding signaling-gateway- 
specific command line interface command, "REPT-STAT- 

25 SLK:LOC=1 101 :PORT=A." Functionally similar abstract command language 
commands associated with different devices may utilize the same root 
command with different arguments. For instance, a communication interface 
package associated with a service control point may include the following 
link alignment status abstract command language command, 

30 u lsLinkAligned(L5), n where L5 is a signaling link connected to the service 
control point. As such, different devices can share a command structure that 
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is common to a large degree, and highly intuitive from an operator's 
perspective. 

Referring back to Figure 4, communication interface packages may 
be stored in communication interface package library 356 that is maintained 
5 in version-controlled environment 350. As discussed above, communication 
interface packages may include software classes created using an object- 
oriented programming environment, such as C++ or J++. Consequently, a 
new communication interface package object can be created which inherits 
some or all of the properties associated with a parent communication 
10 interface package object. This inheritance capability facilitates the rapid 
creation of communication interface packages as new devices and new 
versions of existing devices are added to a test system. Communication 
interface packages may be included and compiled in text cases using 
INCLUDE statements or other statements that allow files to be incorporated 
1 5 into the compiled version of a test case. 

Test plan and test case editor 314 allows a user to create and edit 
individual test cases as well as high-level test plans. In the most simplistic 
embodiment, a test case is a text file that contains a sequence of test 
instructions for testing one or more devices under test. A test plan is a 
20 sequence of test cases. Such test instructions may include abstract 
command language and tool command language commands that perform 
specific operations on a DUT. These instructions may also facilitate 
conditional processing based on the results returned from one or more 
devices under test. In addition, multiple test cases can be combined into a 
25 single test case data file and thereby provide test-plan-like functionality. A 
sample test case is shown below: 
package require Mgts 
package require Eagle 
package require titen 



30 
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# This is the startup code that would be run before any test 

# purposes are started. 

proc startup {} { 

Mgts mgtsl $::env(MGTS_SHELF) $::env(MGTS_BUILD) 
Eagle eaglel $::env(EAGLEJP) $::env(EAGLE_PORT) 
eagle login $::env(EAGLE_LOGIN) 

$::env(EAGLE_PASSWORD) result 

} 



# 

# This is the cleanup code that is run after all the test purposes 

# have completed execution, 
proc cleanup 0 { 

15 Mgts::deleteAII 

Eagle::deleteAII 

} 

# Test Purpose #1 
20 step tp { 

Purpose - ANSI Level 3 AQTest7.2.4 

}{ 

set nodename $::env(L3_NODE_NAME) 
25 set L1_1_card $::env(L1_1_CARD) 

set L1_1 j3ort $::env(L1_J_PORT) 
infoline "Start the State Machine. L3 Node Name is 
$nodename" 

mgtsl runSM $nodename AQTest7-2-4 



Mgts ::expectE vent { 
MGTSStateTransition(V,\UserActton) { 
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# Got user action 

infoline "Inhibiting link L1_1" 
break 

} 

MGTSStateTransition(*,*,*,NotAligned){ 

# The State Machines did not align. *DO NOT* continue 
infoline "Transitioned through NotAligned state!" 
result ABORTED 

return 

} 

} 

# Now, inhibit link L1_1 

eaglel lnhibit_Link(L1,1) 

if {![Eagle::isOK result]} { 

infoline "Eagle rejected command, check LOC and PORT 
variables $result(command)" 
result ABORTED 
return 

} 

Mgts::expectEvent { 

MGTSStateTransition(*,*,*,FAILED){ 

# One of the State Machines failed. *DO NOT* continue 
infoline "Transitioned through FAIL state!" 

result FAIL 
return 

} 

MGTSStateTransition(*,*,*,PASSED){ 
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# State Machines Passed 

infoline "Transitioned through PASS state" 

result PASS 

return 

5 } 

MGTSStateTransition(* t *,UserAction,Stop) { 
infoline "Transitioned through Timeout state! Failing 
testcase." 

result FAIL 
10 return 

} 



} 

} 

# Start execution 
run testcase 



In the code example, the "package require" statements include device 
20 packages required for the script. In this example, the device specific 
packages that are included are mgts, Eagle, and Titen. According to an 
important aspect, the Titen package includes procedures that allow the user 
to create test cases using generic commands. The procedures access the 
device specific packages for multiple devices being tested and perform the 
25 functions for each specific device. For example, the "proc startup{}" 
procedure includes code for starting up and logging into each device. The 
procedure also creates an instance of an object for each device. For 
example, in the statement "MGTS mgtsl," "MGTS" creates an instance of an 
MGTS object having an identifier of "mgtsl." The remaining statements in 
30 the "proc startup{}" procedure control the login and password for the Eagle 
device. Thus, all the user is required to do in writing a test script is specify 
the command "proc startupQ" and the devices that the user desires to start. 
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The code in the "proc_startup{} procedure accesses the device packages for 
the individual devices and performs the functions, such as initializing 
memory, bringing links into service, aligning links, etc., for each specific 
device. 

5 The "proc cleanup{}" procedure includes cleanup routines for each 

device that are automatically called at the end of the script. Like the 
a startup{} n procedure, the cleanup procedure is a procedure in the titen 
package that accesses the individual device packages. The M cleanup{}" 
procedure performs device-specific cleanup functions, such as freeing 

10 memory, disabling links, etc. Again, all the test case designer is required to 
remember is the "cleanupO" command and the names of the devices that the 
designer desires to shut down. Thus, the titen package reduces the time 
and effort required for test case design. 

The "step tp n statement defines a test in a test case. The steps are 

1 5 counted automatically by the Titen package and a result for each step will be 
generated if using TET/API-compliant test cases. 

The "purpose" statement is a free form text field describing a test 
case. This statement can be a block of text that describes the execution of a 
test. 

20 The "set" statements set environment variables read into the test. 

The "mgtsl runSM" statement runs the name or number of the 
desired state machine on the specified node. In this example, the state 
machine is "AQTest7-2-4. n 

The "Mgts::expectEvent n procedure looks for the state machine to 
25 advance to a particular state in the PAS M map. The UserAction state looks 
for this state in PASM and indicates that a used action on the device under 
test is needed. 

The "InhibiMJnk" command is an abstract command language 
command which may be converted into a device specific command, such as 
30 eagle cli "INH-SLK:LOC=$L1_1_card:PORT=$L1_1_port". In this example, 
the abstract command language command is converted into an Eagle- 
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specific command including specific variables to inhibit a signaling link of an 
Eagle STP. 

The "Mgts::expectEvenf procedure looks for the PASM state 
transition after the Eagle command was sent. In particular, the procedure 
5 looks for FAILED, PASSED, or a transition from a userAction state to Stop 
for a timeout or abort. 

The M run_testcase" statement is a wrapper that calls execution of the 
test case. 

Test plan and test case editor 314 communicates and manages 

10 interaction with the version-controlled test plan and test case library 352. As 
such, individual test cases and test plans are checked in and out of version 
control environment 350 via editor 314. Editor 314 is also adapted to 
interface with the graphical user interface 310, thereby providing users with 
an easy to use and intuitive file creation/modification environment. 

15 As stated above, test tools server 210 includes a scheduler that 

allows the user(s) to schedule immediate or delayed execution of individual 
test cases or test plans. Such test case and test plan scheduling capability 
enables test operators to efficiently allocate high-demand resources in a test 
lab environment by scheduling the execution of a test plan or test case 

20 during off-hours or periods of low resource usage. In one embodiment, the 
scheduler allocates resources by examining test environment requirements. 
In the ideal situation, the test environment only specifies the type of resource 
the test requires, and it is the responsibility of the scheduler to allocate the 
necessary resource(s) prior to test case execution. This approach works 

25 best for environments with a large number of uniform resources. Such a 
typical case would be an E-commerce server where the resource is the login 
ID of a particular customer or large-scale telecommunications equipment 
where the test is restricted to a particular shelf or even a card within a shelf 
and no special wiring is required (i.e., in an environment where the ratio of 

30 resources required by a test case to total number of generic resources is 
very low). On the other hand, if the number of resources used by a test case 
is close to the total amount of available resources, it may be more efficient to 
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schedule on per-resource basis. Such is often the case in a typical 
telecommunications lab where equipment cannot be used to run more than 
one test case at any time, and there are only a few units available. In such a 
case, the resource schedule is defined as a relation between instances of 
5 resources, tester, test suite and time interval. Resource schedules are 
stored in a resource schedule repository accessible by the scheduler. 

In any event, manager 316 receives test plan and/or test case 
execution instructions from a test operator via GUI 310 and subsequently 
extracts or reserves the required test plan or test case files from test plan or 

10 test case library 352 and communication interface package files from 
communication interface package library 356, both of which are maintained 
in version control environment 350. Once all appropriate test plan, test case 
and communication interface package files have been obtained from version 
control environment 350, the indicated test or tests are executed. The 

15 results of a particular test plan or test case execution are subsequently 
stored in a test results library 354, which is also maintained in version- 
controlled environment 350. 

In one embodiment, test plan and test case manager 316 examines 
information contained in a test case or test plan related to the operating 

20 software version required on a particular test device. Once such operating 
software version information is determined, manager 316 communicates with 
a device under test operating software library 358 contained in the version 
control environment 350. DUT operating software library 358 maintains 
software associated with particular test devices (e.g., generic program loads, 

25 operating systems, feature enhancements, etc.), which enables a test 
system of the present invention to automatically and dynamically configure a 
DUT using any number of pre-defined software loads. As such, a particular 
version or revision of operating software associated with a DUT can be 
automatically downloaded and initialized on a specific DUT that is involved 

30 with the test plan/test case being executed. 

Reporting manager 318 is extracts some or all of the files associated 
with a test plan or test case from version control environment 350. These 
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files may include test plan or test case files, associated CIP files, test result 
files, and DUT operating software files. As such, reporting manager process 
318 facilitates the presentation of a comprehensive, integrated view of 
information related to a particular test plan or test case including: test plan or 
5 test case instructions, relevant DUT specific command packages, results 
associated with the executed test plan/test case instructions, and operating 
software source code associated with relevant DUT(s). This information is 
accessible and displayed to a test operator via GUI interface 310. 

Test Plan/Case Execution Engine 

10 Figure 6 illustrates one embodiment of test plan and test case 

execution manager 316, including information flow pathways associated with 
a sample system under test. The functional blocks illustrated in Figure 6 are 
intended to illustrate services provided by execution manager 316. Practical 
implementation of this functionality could be accomplished using a variety of 

15 software constructs and architectures, and as such, the block diagram 
shown in Figure 6 should be considered as only one of many possible 
implementation schemes. 

In addition to execution manager 316, Figure 6 includes a system 
under test that includes a signaling gateway node 226, a message generator 

20 and traffic simulation device 228, and a Softswitch 230, The functions 
provided by each of these devices is not particularly relevant to the operation 
of the present invention, and as such detailed descriptions of these devices 
are not provided herein. For the purposes of illustration, each of the devices 
under test are assumed to include a command line interface with which the 

25 execution manager 316 communicates. However, it will be appreciated that 
in an alternate embodiment of the present invention, execution manager 316 
may communicate with a device under test that utilizes a GUI-type interface 
as opposed to a command line interface. In such cases, a GUI-interfaced 
device under test may be configured to serve as a remote host and accept 

30 command inputs from the execution manager 316 via a connection that is 
similar in function to that provided by commercially available remote access 
products such as PCANYWHERE™ available from Symantek Corporation or 
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CARBONCOPY™ available from Compaq Corporation. In an alternate 
embodiment of the invention, remote access can be achieved using any 
TET-enabled remote client, such as TestExpert, available from Lumenar, 
Inc., that supports client/server on the execution host or on a thin client web 
5 browser. The commands and command structures necessary to 
communicate with such GUI driven devices would be accommodated within 
the basic architecture of the present invention via GUI-capable 
communication interface packages. 

Once again, for the purposes of illustration, the detailed discussions 

10 presented herein are limited to communication with those devices that 
employ command line interfaces, since this type of interface is currently the 
most widely deployed interface type. Communication with GUI-based 
devices can occur via a graphical user interface if a suitable GUI tester is 
added via a new package. 

15 Returning to Figure 6, execution manager 316 includes an execution engine 
400, a titen package 402, a plurality of device-specific test case packages 
404, and a communication interface layer 420. Execution engine 400 reads, 
interprets, and executes commands in a test plan or test case file. Titen 
package 402 includes generic classes for executing test cases, recording 

20 results, and performing common functions by accessing individual device 
packages, as discussed above. Packages 406, 408, 410, and 412 include 
device-specific classes. According to an important aspect of the invention, 
these device-specific classes include functions that translate generic 
commands to device-specific commands. In one embodiment, the device- 

25 specific commands may be tool command language commands. In an 
alternate embodiment, the device-specific commands may be implemented 
using an alternate language, such as PERL, JAVA, or C++. The abstract- 
command-language-to-tool-command-language command mapping 
information is obtained for any given DUT from a DUT specific 

30 communication interface package. As described above, Figure 5 illustrates 
exemplary tool command language to DUT-specific commands. 



WO 02/056541 



PCT/US01/48677 



-23- 

In the embodiment shown in Figure 6, a DUT package is included for 
each DUT that is involved in a test. Each device-specific package includes 
device-specific commands and mappings from generic to device-specific 
commands. In the example shown in Figure 6, the device-specific packages 
5 include a signaling gateway package 406, a Softswitch package 408, and a 
message generator/traffic simulator (MGTS™) package 410. 

Titen package 402 is the "glue" software component that pulls all of 
the packages together and provides basic common environment 
functionality. It is the part of the present invention that interfaces to the high- 

10 level TMS to the scripting languages. For example, titen package 402 
handles automatically counting the steps of a test case and assembling them 
into the invocable components required by the test environment toolkit 
(TET). The TET is a universal management and reporting framework for 
conformance tests. Further information regarding TET can be found at 

15 www.tetworks.openqroup.org . Titen package 402 also includes start-up and 
clean-up routines required by the device packages. This allows concise, 
clean, and easy means to handle all device package overhead. Thus, all of 
the setup and clean up on the main script can be done with a couple of lines 
of code. 

20 In one embodiment, titen package 402 provides tlje functions step, 

invocableComponent, infoline, result, run_testcase, try, startup, and cleanup. 
These actions performed by these functions are as follows: 

step: 

25 a command used to define a step in a testcase, which can then be 
automatically counted. 

invocableComponent, subTest: 

a command used to define a new TET invocable component. Each 
30 invocable component contains a set of test case steps. 

infoline: 
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a command used to print output into a journal file, 
result: 

a command used to report the result of a step. If no result is provided, it 
5 defaults to NoResult. 

run_testcase: 

a command that is a wrapper to call for execution of the testcase 
10 try: 

a command similar to JAVA'S "try" command that evaluates the expressions 
given 

startup, cleanup: 

1 5 initialization and cleanup for execution of packages 

The command set listed above are procedures defined in the titen 
package. Some of the procedures, such as startup and cleanup, greatly 
reduce test case complexity because they include code that accesses the 
individual device packages for performing startup and cleanup functions. 

20 For example, if a user specifies "startup eaglel , routerl , w the startup 

procedure in the titen package automatically accesses the STP and router 
packages, and determines how to start each device. The test case designer 
need not be familiar with individual device commands or startup procedures. 
Allowing actions to be performed on multiple devices using a single generic 

25 command is an important feature of the invention. 

In the embodiment shown in Figure 6, communication interface 420 
automatically establishes a telnet session with the command line interface of 
each device under test. The present invention is not limited to using telnet to 
connect to the command line interface of the device under test. Any protocol 

30 that allows remote terminal access to a machine is intended to be within the 
scope of the invention. In any event, communication interface 420 is further 
adapted to send and receive DUT-specific command line interface 



WO 02/056541 



PCT/US01/48677 



-25- 

commands and receive DUT-specific results. As indicated in Figure 6, such 
commands are generated by execution engine 400, while DUT test results 
are generated by the various devices that comprise the system under test 
(i.e., SG 226, SS 230, and MGTS™ 228). 
5 Test Plan and Test Case Execution 

Continuing with the example test scenario presented in Figure 6, SG 
226 and MGTS™ 228 communicate via signaling system 7 (SS7) signaling 
links, while SG 226 and SS node 230 communicate via stream control 
transmission protocol/Internet protocol (SCTP/IP). It should be noted that 

10 the test system of the present invention is not concerned with nor adapted to 
communicate with test devices via such inter-nodal communication 
interfaces. Instead, as discussed above, the present invention 
communicates with and controls devices under test via their administrative 
interfaces (e.g., command line, GUI). 

15 Figure 7 is a process flow diagram that describes exemplary steps 

associated with execution of a test case according to an embodiment of the 
invention. Corresponding information flow pathways are indicated in Figure 
6. In the description below, it is assumed that a particular test case has 
been selected for execution by a test operator via a TMS client 214. Test 

20 tools server schedules the test for execution and when the time for execution 
arrives, extracts the test from library 352 and downloads the test to 
execution engine 400(ST1). Execution engine 400 opens the file and begins 
reading the file. Information contained within the test case file identifies one 
or more of the devices under test. Execution engine 400 accesses 

25 communication interface package library 356 and extracts the appropriate 
communication interface packages associated with each DUT, as indicated 
in step ST2. Information may also be included in the test case file that 
identifies particular DUT operating software requirements associated with 
the test, in which case execution engine 400 accesses DUT software library 

30 358 and directs the appropriate software loads to each DUT (ST3). 

Once initialization of all DUTs is complete and all appropriate DUT 
objects and instances have been created, test command instruction 
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processing begins. The execution engine 400 reads an abstract command 
language command (ST4) and, based on the mapping provided by the 
appropriate communication interface package, interprets the command 
within the context of the specific DUT to which the command refers (ST5). 
5 For instance, the abstract command language command 
"IsLinkAlignedfLI.I)," when directed to SG 226, would require input from SG 
package 406. The SG-specific communication interface package that was 
obtained from CIP library 356 contains the abstract-command-language-to- 
tool-command-language mapping information necessary for execution 

10 engine 400 to successfully interpret the command and produce an 
equivalent tool command language command (ST6). It should be noted that 
the mapping of abstract command language commands to tool command 
language commands does not necessarily yield a one-to-one 
correspondence. That is, a single abstract command language command 

15 associated with an SG node may map or translate to a multi-command tool 
command language sequence. The same property holds true for tool 
command language to abstract command language command mapping. 
That is, multiple abstract command language commands may correspond to 
a single tool command language command. Thus, the present invention is 

20 not limited to a one-to-one between abstract command language commands 
and tool command language commands. 

In any event, the resulting tool command language command is 
subsequently passed to the communication interface 420 (ST7), as indicated 
in Figure 6. In one embodiment, communication interface 420 contains 

25 components that include commonly available software tools, such as the 
Expect tool, available from http://expect.nist.gov. Software tools, such as 
Expect, provide the communication interface 420 with the ability to establish 
telnet sessions with various devices under test, and also provide a means for 
capturing streams of response data (e.g., ASCII character data) from a DUT. 

30 In the case of response data, once captured this data is provided to 
execution engine 400 for subsequent processing. 
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As such, the outbound command is transmitted via a telnet session to 
the appropriate DUT (ST8). An example of one such outbound signaling 
gateway command 372 is shown in Figure 8. Command 372 illustrated in 
Figure 8 requests the status of a signaling link connected to signaling 
5 gateway 226. Communication interface 420 may subsequently enter a "wait" 
mode, during which interface 420 monitors the telnet session and waits for a 
response from the DUT (ST9). Once again, a typical DUT response is 
comprised of a stream of ASCII characters that form a text message which 
was originally intended to be read by a human operator via the DUTs 

10 administrative interface. Such DUT response information might include test 
result information, device configuration information, device status 
information, an acknowledgment message associated with a received 
command, etc. An example of one such response message 374 is shown in 
Figure 8. In Figure 8, response message 374 indicates a state transition for 

15 a signaling link from active to aligned. 

In one embodiment, communication interface 420 intercepts the 
response data stream and provides execution engine 400 with the DUT 
response message in ASCII text format. Execution engine 400 receives the 
DUT response message, parses, and examines the text message (ST11). 

20 The ability to receive and parse DUT response messages without human 
operator intervention is one of the key aspects of the present invention. 
Such functionality enables test case scripts to be created that are capable of 
employing branching or conditional code structures that effectively enable 
dynamic test sequences to be executed which vary depending on the 

25 response of a DUT to a previous command (ST12 and ST13). For example, 
a test case can be created which directs MGTS™ device 228 to attempt to 
establish or align an SS7 signaling link that would be connected to SG node 
226. The test case code could be structured such that if a response 
message is received from MGTS™ 228 indicating that the link could not be 

30 successfully aligned then a second link alignment should be attempted. 
However, in the event that a response message is received from MGTS™ 
228 indicating that the link was successfully aligned then the MGTS™ 228 is 
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directed to begin generating and transmitting a sequence of test SS7 
message signaling units (MSUs) to SG 226. 

An even more powerful extension of this particular test system 
functionality involves the ability of a first DUT to control a second DUT, even 
5 though the first DUT has no knowledge of the second DUT's control or 
administrative interface protocol. Such functionality is accomplished in much 
the same manner as is described in the previous example, only in this case 
the branching or conditional construct in the test case code involves multiple 
DUTs. 

10 For example, a test case can be created which directs MGTS™ 

device 228 to attempt to establish or align a first SS7 signaling link that 
would be connected to SG node 226. The test case code could be 
structured such that if a response message is received from MGTS™ 228 
indicating that the first link could not be successfully aligned then another 

15 alignment attempt on the first link should be initiated. However, in the event 
that a response message is received from MGTS™ 228 indicating that the 
first link was successfully aligned, then SG 226 is subsequently directed to 
drop or terminate a second signaling link. As such, MGTS™ 228 effectively 
directs SG 228 to initiate a link termination action. 

20 The following code example illustrates a portion of a test case in 

which execution engine 400 simultaneously tests multiple devices. In this 
example, the first device under test is an STP, referred to as "eagle." The 
second device under test is an IP router, referred to in the code as "routerl 
Finally, a message generator/traffic simulator device M mgts1 B is used to 

25 perform various tests on the devices under test. The line numbers 
appearing on the right hand side of each line of code are included for 
illustrative purposes and will be used to explain the functions of various lines 
of code. In the lines of code, the statements that begin with # signs are 
comments. 

30 1 # deactivate SS7 link and verify alarm 

2 # else deactivate IP router port and verify alarm in 3 sec 
3 
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4 mgtsl cli Deactivate link: Port= $SSP1_port 
5 

6 # check for alarms 

7 set time 0 

5 8 while {$time < 3} { 

9 eagle cli "REPT-STAT-ALM" result 

10 if {[regexp "ALARM. *MAJR" [lindex $result (response) 
1]]} { 1 1 set eagle_alarm 1 

1 2 infoline "Eagle alarm verified" 

10 13 break 
14 

15 }else{ 

16 # deactivate an IP Router port" 

17 routerl cli Deactivate port: Port= $SSP2_port 
15 18 infoline "Router IP port disabled" 

19 } 

20 sleep 1 

21 set time [expr $time +1] 



20 



22 } 



In line 4 of the code, execution engine 400 deactivates SSP port 1 on 
the MGTS. In a real network, deactivation of a port from an SSP should 
trigger an alarm at the corresponding STP. Accordingly, in line 8 of the 
code, the test case includes a while loop that checks for an alarm at the 

25 eagle STP. In line 9 of the code, the test case checks for the alarm status of 
the eagle. In line 10, the code checks for whether there is a major alarm on 
the eagle. In line 11, the code sets the eagle alarm flag if a major alarm 
occurs. Line 12 of the code logs the alarm in a log file. I 

In line 17, the code deactivates SSP port 2 on an IP router. In line 18, 

30 the code logs the fact that the IP router port was disabled. The code then 
returns to the beginning of the while loop and checks whether the Eagle 
detects the fact that the IP router port to the SSP was disabled. Thus, in the 
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code example, the execution manager performs a conditional action based 
on data received from a device under test. This is a feature of the invention 
that has not previously been possible in conventional test systems. In 
conventional test systems, a human operator would be required to enter the 
5 command for deactivating a link via the command line interface of one 
device, monitor the results on a separate terminal, decide what action to take 
based on the results, and manually reconfigure the second device. 
Accordingly, it can be seen that the present invention can save time and 
effort when testing multiple different devices. 

10 It will be appreciated that significantly more complex multi-DUT 

interactions can be easily accommodated by a test system of the present 
invention. For instance, SS7 level 2 and level 3 conformance testing is 
currently a complex and time consuming activity that often takes days or 
weeks to complete by a human operator. A test system of the present 

15 invention can be easily configured to automatically administer complete SS7 
level 2 and level 3 conformance tests within a matter of hours, without the 
need for human operator intervention. 

It will be understood that various details of the invention may be 
changed without departing from the scope of the invention. Furthermore, the 

20 foregoing description is for the purpose of illustration only, and not for the 
purpose of limitation-the invention being defined by the claims. 
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[received by the International Bureau on 13 December 2002 (13.12.2002); 
original claims I, 13, 19 and 32 amended; new claim 39 added; 
remaining claims unchanged (6 pages)] 

What is claimed is: 

1. A method for testing a system of communications network devices, 
5 the method comprising: 

(a) reading an abstract command language command from a test 
case and a device-specific object corresponding to a type of 
telecommunications network node, the device-specific object 
having a function named according to the abstract command 

10 language command, the function including device-specific 

commands compatible with a command line interface of the 
type of telecommunications network node; 

(b) translating the first abstract command language command into 
a device-specific command compatible with an administrative 

15 interface associated with a first device under test (DUT) using 

the function defined in the first object; and 

(c) communicating the device-specific command to the first DUT. 

2. The method of claim 1 comprising receiving a reply from the first DUT, 
and, in response, automatically generating a second device-specific 

20 command compatible with an administrative interface associated with 

a second DUT. 

3. The method of claim 2 wherein receiving a reply message includes 
receiving a message containing test result information. 

4. The method of claim 2 wherein receiving a reply message includes 
25 receiving a message containing an acknowledgment message. 

5. The method of claim 2 wherein receiving a reply message includes 
receiving a message containing status information associated with the 
first device under test. 

6. The method of claim 2 comprising generating a second abstract 
30 command language command in response to the reply message and 

translating the second abstract command language command into the 
second device-specific command. 
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7. The method of claim 2 wherein generating a second device-specific 
command includes generating a command line interface command 
compatible with a command line interface of the second DUT. 

8. The method of claim 2 wherein the second DUT is a packet network 
5 testing device. 

9. The method of claim 1 wherein translating the abstract command 
language command includes accessing a device-specific 
communication interface package that contains abstract command 
language translation instructions. 

10 10. The method of claim 1 wherein translating the abstract command 
language command into a device-specific command includes 
translating the abstract command language command into a 
command line interface (CLI) command compatible with a command- 
line interface associated with the first DUT. 

15 11. The method of claim 1 wherein communicating the device-specific 
command to the first DUT includes transmitting the device-specific 
command via a telnet session. 
12. The method of claim 1 wherein the first DUT is a packet routing 
switch. 

20 13. A method for dynamically testing a system of communications 
network devices, the method comprising: 

(a) reading a first abstract command language command 
associated with an operation involving a first device under test 
(DUT) and a device-specific object corresponding to a type of 

25 telecommunications network node, the device-specific object 

having a function named according to the abstract command 
language command, the function including device-specific 
commands compatible with a command line interface of the 
type of telecommunications network node; 

30 (b) translating the first abstract command language command into 

a first command line interface command using the function 
defined in the first object; 
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(c) transmitting the first command line interface command to the 
first DUT; 

(d) receiving a reply message from the first DUT; 

(e) dynamically selecting a second abstract command language 
5 command based on contents of the reply message; and 

(f) executing the second abstract command language command. 

14. The method of claim 13 wherein translating the first abstract 
command language command into a first command line interface 
command includes accessing a communication interface package 

10 associated with the first DUT. 

15. The method of claim 13 wherein transmitting the first command line 
interface command to the first DUT includes transmitting the 
command line interface command using remote terminal access 
software. 

15 16. The method of claim 1 3 wherein transmitting the first command line 
interface command to the first DUT includes transmitting the 
command line interface command over a wide area network (WAN). 
17. The method of claim 13 wherein receiving a reply message includes 
receiving test result information. 

20 18. The method of claim 13 wherein receiving a reply message includes 
receiving an acknowledgment message associated with a previously- 
executed command. 
19. A system for testing communications network devices, the system 
comprising: 

25 (a) a device-specific communication interface package including 

an object corresponding to a type of telecommunications 
network node, the object including functions named according 
to abstract command language commands, the functions 
including commands compatible with the type of 

30 telecommunications network node; 

(b) an execution manager for translating abstract command 
language commands to device-specific commands using the 
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functions defined in the device-specific communication 
interface package; and 
(c) a user interface for initiating the execution of an abstract- 
command-language-based test procedure and displaying 
5 subsequent test results. 

20. The system of claim 19 wherein the device-specific command is a 
command line interface command. 

21 . The system of claim 19 wherein the execution manager is adapted to 
receive and interpret a response message generated by a DUT. 

10 22. The system of claim 19 wherein the execution manager is adapted to 
direct a DUT to be configured with a specific operating software 
version. 

23. The system of claim 19 wherein the execution manager is adapted to 
schedule the execution of a test case or a test plan. 
15 24. The system of claim 19 wherein the user interface is a graphical user 
interface (GUI). 

25. The system of claim 19 wherein the user interface is adapted to 
simultaneously display related test plan, test case, and test result 
information. 

20 26. The system of claim 19 including a version-controlled environment for 
maintaining information associated with a test and a device under test 
(DUT). 

27. The system of claim 26 wherein the version-controlled environment is 
adapted to maintain DUT-specific communication interface packages. 
25 28. The system of claim 26 wherein the version-controlled environment is 
adapted to maintain test plan and test case files. 

29. The system of claim 26 wherein the version-controlled environment is 
adapted to maintain test result information. 

30. The system of claim 26 wherein the version-controlled environment is 
30 adapted to maintain DUT operating software. 
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31. The system of claim 19 including a test case editor for editing test 
case information, wherein the test case editor is accessible via the 
user interface. 

32. A system for simultaneously testing a plurality of communications 
5 network devices, a test tools server for maintaining test cases and for 

storing the system comprising: 

(a) a test tools server for storing a plurality of device-specific 
communication interface packages, each device-specific 
communication interface package including an object 

10 corresponding to a type of telecommunications network node, 

the object including functions being named according to 
abstract command language commands and having command 
line interface commands corresponding to the type of 
telecommunications network node; and 

15 (b) a test management system client for requesting execution of 

test cases on the devices under test; and 
(c) a test controller for receiving the test cases and the 
communication interface packages, connecting with a plurality 
of communications network devices, and executing the test 

20 cases to test the network communications devices. 

33. The system of claim 32 wherein the test tools server stores a generic 
communication interface package containing common procedures for 
accessing the device-specific packages and performing device- 
specific functions during a test case. 

25 34. The system of claim 32 wherein the test tools server includes an 
arbitration engine for dynamically resolving test case scheduling 
conflicts. 

35. The system of claim 33 wherein the generic communication interface 
package includes a startup procedure for accessing the devtce- 

30 specific packages to initialize each device involved in a test case. 

36. The system of claim 33 wherein the generic communication interface 
packages includes a cleanup procedure for accessing the device- 
specific packages to free resources on each device involved in a test 
case. 
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37. The system of claim 32 wherein the device-specific communication 
interface commands include functions for mapping abstract command 
language commands to tool command language commands. 

38. The system of claim 32 wherein the test controller is adapted to 
5 simultaneously communicate with command line interfaces of the 

communications network devices in order to execute test cases and 
record results. 

39. An abstract command language for testing telecommunications 
network components, the abstract command language including 

10 computer-executable instructions embodied in a computer readable 

medium, the abstract command language comprising: 

(a) a plurality of objects, each object corresponding to a type of 
telecommunications network node; and 

(b) a plurality of functions associated with each object, each 
15 function being named according to an abstract command 

language command for testing one of the types of 
telecommunications network nodes, each function including 
command line interface commands specific to one of the types 
of telecommunications network nodes. 
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